DocEdit: Language-Guided Document Editing

Professional document editing tools require a certain level of expertise to perform complex edit operations. To make accessible increasingly novice users, we investigate intelligent assistant systems that can or suggest edits based on user's natural language request. Such system should be able understand the ambiguous requests and contextualize them visual cues textual content found in image localized unstructured text structured layouts. this end, propose new task language-guided editing, where user provides an open vocabulary request, produces command used automate real-world software. In support task, curate DocEdit dataset, collection approximately 28K instances over PDF design templates along with their corresponding ground truth software executable commands. our knowledge, is first dataset diverse mix operations direct indirect references embedded objects such as paragraphs, lists, tables, etc. We also DocEditor, Transformer-based localization-aware multimodal (textual, spatial, visual) model performs task. The attends both related contents which may referred generating embedding predict associated bounding box localizing it. Our proposed empirically outperforms other baseline deep learning approaches by 15-18%, providing strong starting point for future work.